ivory tower (n.)

symbol of artistic or intellectual aloofness, by 1889, from French tour d'ivoire, used in 1837 by critic Charles-Augustin Sainte-Beuve (1804-1869) with reference to the poet Alfred de Vigny (1797-1863), whom he accused of excessive aloofness.

Et Vigny, plus secret, comme en sa tour d'ivoire, avant midi rentrait. [Sainte-Beuve, "Pensées d'Août, à M. Villemain," 1837]

Used earlier as a type of a wonder or a symbol of "the ideal." The literal image is perhaps from Song of Solomon [vii:4]:

Thy neck is as a tower of ivory; thine eyes like the fishpools in Heshbon, by the gate of Bathrabbim: thy nose is as the tower of Lebanon which looketh toward Damascus. [KJV]